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This document reports on a study of the feasibility 
cf a procedure for evalueiting an instructional program. The 
instructional units evaluated were professional education courses 
which were structured around performance objectives and utilized a 
model similar to the Personalized System of Instruction (PSI) . Data 
were collected in four phases; instruments used included the 
California Psychological Inventory^ the Tennessee Self-Concept Scale^ 
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experience. Two hundred eighty-one variables resulted for each 
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to four types of regression analyses; the analyses were interpreted, 
indicating that the goals of the project were met and a decision 
equation was derived. (SD) 
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Abstract 

This project was conducted to mathematically integrate longitudinal 
data obtained from secondary teachers into a curriculum decision procedure. 
A four phrase data collection procedure was implemented into a secondary 
level teacher preparation program to provide the variables for assessing 
the quality of various program components. Regression analysis was 
utilized to determine the relation between criterion and independent 
variables related to two of the nine instructional units. Regression models 
accounting for 44% and 68.5% of the variance were found to occur for the 
self-analysis and performance objectives instructional components, 
respectively. Further analyses provided numerical values to determine the 
effectiveness of each instructional unit. 
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Comprehensive curriculum evaluation is a very complex process. This 

assertion is reasonable for no other reason th^rT the variety of variables, 

such as the fol lowir;,, that must be attented to: learner achievement, 

interpersonal coimiunications , pov^er structures, school organizational 

climates, variety and quality of instructional resources available, and 

societal norms concerning education. With such a variety of concerns 

vying for attention, an evaluation project must define evaluation and 

differentiate the goal from the functions if the evaluation process is 

to contribute to the curriculum under scrutiny. In an oft-cited work, 

Scriven ( 1967) described evaluation as a methodological activity in tlie 

fol 1 o^'n* ng manner. 

"The activity consists simply in tlie gathering and 
corrbining of performance data vn'th a weighted set 
of goal scales to yield either comparative or 
numerical ratings and in the justification of (a) 
the data gathering instruments, (b) tfie weighting 
and (c) the selection of goals." (p 40) 

Expanding on this definition, Scriven stated that evaluaticri should not 

only collect and analyze data, but should make judgn)ents and report 

these judgments publicly. 

Another significant definition of evaluation was developed by the 

Phi Delta Kappa National Study Comr.rittee on Evaluation (Stuf f 1 ebeam, 

Foley, Gephart, Guba, Hammond, Herriman, Provus, 1971): 

"Evaluation is the process of deline-^.ting, obtaining 
and providing useful information for judging decision 
alternatives." (p XXV) 

In contrast to Scriven, the authors of this definition emphasize that 

evaluation is a continuing process which provides information that should 

guide decision making, not produce judgments. The authors of these 

respective definitions agree that evaluation is a process whereby data 
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are gathered for the purpose of decision making, but disagree on who is 
to make the decisions. The issue of who makes the decisions may not 
be central if the scope of the curriculum evaluation is limited to a 
small scale project and the curriculum developers are also serving as 
evaluators. When curriculum developnient and subsequent evaluation of 
the curriculum occurs in educational settings' whether at the local 
district or building level, or wittiin higher education at the college or 
department level, the developer and evaluator roles may be assumed by 
^e same individuals. It is uSSumGd that under tfiese circumstances 
the curriculum is being designed for a particular educational setting 
with no grandiose plans for marketing the curriculum regionally or 
nationally. Under these conditions, decision making in regard to tite 
effectiveness of instructional components is a viable goal and violates 
neither Scriven's nor the PDK coiiMiittee's formulations of the concept 
"evaluation." 

With an evaluation goal in mind, attention must be directed to the 
functions of the evaluation process. These functions may be directed 
either to the developments execution, and implementation of a curriculum 
or to the political and economic support for the curriculum (Zais, 1976, 
p 377). Because of the different emphases that are" possible, it is diffi- 
cult to develop a single generalized model for curriculum evaluation. 
On a positive note however, numerous conceptual models outlining various 
types of evaluation have been advanced during the past decade. For 
example, the Countenance Model (Stake, 1957), Formative and Suninative 
Evaluation (Scriven, 1967), the Modus Operandi Method (Scriven, 1974), 
the CIPP (Content, Input, Process, Product) Evaluation Model (Stuf flebeam, 
et al , 1971), the Discrepancy Model (Provus, 1971), and the Center for the 
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Study of Evaluation (CSE) Model (Alkin, 1974) are among the more familiar 
models • These models identify critical decision-making points along the 
continuum of processes occurring in curriculum developn)ent, particularly, the 
development sequence championed by Tyler (1950) and sustained by Taba (1962). 

Once a model is selected or created from the theoretical constructs 
provided by the various models, the evaluator is faced with pragmatic 
issues of identifying appropriate instrumentation, selecting the sample, 
and analyzing the data. In his definition of eval uation, Scriven refers 
to the issue of combining data with weighted goal scales to produce 
numerical ratings and to the justification of those "weightings." Ttiis 
weighting construct is intriguing and should influence data analysis 
significantly. Weighting the data sources in terms of their relative 
importance prior to data collection appears to be what Scriven is 
suggesting when he discusses the primary, secondary, and tertiary effects 
of the curriculum on the various actors affected by the curriculum. 
The effects or the materials on the learners' mental and nonmental 
abilities and attitudes are labeled as the primary effects of the 
curriculum. Secondary effects of the curriculum affect those individual's 
who implement the curriculum, namely, teachers, teacher aides, super- 
visors, while tertiary effects are those effects on the school or other 
students brought about by learnei^s or teachers who exiiibit the primary 
or secondary effects (Scriven, 1957, pp 74-82). 

If primary effects such as achievement data, attitude data, and 
subsequent follow-up information are obtained from learners, should all 
of these data sources have equal weightings? Moreover, if secondary 
effects, such as supervisor and/or principal ratings are collected and 
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combined with the primary effects, what "weights" should these data 
sources assume? .Rather than assign decision weights a priori , this pro- 
ject: was Ur; taken to develop a procedure whereby various data were 
collected, treated to determine decision weights, then combined in a 
mathematical decision equation. Specifically, the development and 
implementation of a procedure to empirically weight the data was an 
ancillary goal of this project, while the primary goal was the mathematical 
integration of weighted data to evaluate the quality of instructional 
components in professional education coursework. 

Data Source 

The institutional setting for this project was a Land Grant 
University accredited by the Southern Association of Colleges and 
Universities and the National Council for the Accreditation of Teacher 
Education. Tliirty-five secondary level teachers who had recently com- 
pleted supervised student teaching at the secondary level and the 
related professional education coursework constituted the final sample 
for tin's project. The diminutive nature of this sample would be of 
greater concern had the goals of this project been otiier than to illustrate 
the feasibility of an evaluation procedure. 

The instructional units eval uated were components of the professional 
education coursework which immediately preceded student teaching. Six 
semester hours of credit were awarded to the candidate for successfully 
completing the eighteen instructional units contained in tliis coursework. 
Each instructional unit was structured around performance objectives, 
with seventy-seven objectives existing for the total program. Topics 
including discipline, performance objectives, test construction, 



Mathemati cal Equations. . . 5 



interpersonal communication, unit planning, teacher/iaw, teaclier self 
analysis, instructional modes, and individualized instruction were 
among the topics examined. 

The instructional model employed during this coursework enabled 
candidates to remediate and retest objectives not achieved on the first 
assessment. Generally, this instructional model embraced the tenets of 
the Personalized System of Instruction (Keller, 1968) with remediation 
consisting of self-study and tutorial assistance available from vie 
course instructor. Grade credit for the coursework was awarded to the 
candidate on the basis of criterion performances on the various unit 
tests, regardless of whether remediation and retesting were necessary. 

Data Collection 

A four phase collection plan was developed and gradually implemented 
to provide the data necessary for assessing the quality of the program. 
Phase one consisted of personality measures, namely, California 
Psychological Inventory (1965), Tennessee Self-Concept Scale (1964), 
being administered to teaching candidates before instruction commenced. 
Based on results from earlier efforts, the investigator felt various 
subscales of these measures would serve as indicators of candidate 
success in the various instructional components (Denton and Stenning, 1973). 

Monitoring teaching candidate progress during the professional 
education coursework was the primary function of phase two of the plan. 
Information concerning the n;ii;:ner of trials necessary to reach mastery 
for each objective, the nui^ib?r oi objectives actually mastered, and 
student perceptions of the quality of the instructional components were 
determined at the conclusion of each component (Denton and Benr.ett, 1975). 
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The5;e data provided diagnostic information for adjusting each tGactiing 
candidate's instructional program an'd simultaneously provided the 
dependent variable for regression procedures used for quality control 
determination of the curricular components. 

Student teaching performance was the focus for phase tliree of tlie 
collection pU.n. Included in this phase was a procedure to assess student 
mastery of teaching behaviors related to the ob. ectives contained in 
the instructional modules completed prior to student teaching. Formal 
evaluation forms v^ere used which permitted the professional staff, i.e. 
cooperating teacher and university supervisor, to rank and record student 
teacher performance on classroom behaviors and instructional skills. 
These behaviors were directly related to the enabling objectives in 
the general methods and curriculum courses. Information on the degree 
of knowledge transfer into observable te"^.ching behaviors provided tlie 
instructional staff with another measurement of the effectiveness of thp 
instructional components that could potentially be conbined with other 
data to render curricular decisions concerning the preparation program. 

Phase four of the plan requested first year graduates to provide 
their perceptions of the preparation program. Questionnaires were 
supplied to recent graduates after one semester of teaching. This 
follow-up procedure permitted the collection of perceptions of first 
year teachers, tempered by actual classroom experiences, regarding tlie 
quality and value of the instructional components in tlie preparatory 
program (ftosser and Denton, 1977). 

A continuum divided into five intervals was the basic fomat for 
the rating scales and perception instruments employed throughout phases 
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three and four of the collection plan. Tlie uniformity of item format 
among these scales resulted in comparable values, ttiercby eliminating 
the need to trans form raw scores. Descriptive validity of these instru- 
ments was addressed by comparing the items on eacli instrument with the 
performance objectives in the coursework being evaluated. A logical 
relation among the items on each instrument and to one or more performance 
objectives was required for the item to be retained. In addition, a 
reliability estimate, the Cronbach alpha coefficient of .89, was 
determined for the follow-up instrument. 

D ata Ana l ysis 

Two instructional components, namely, performance objectives and 
self analysis skills were selected to demonstrate the four statistical 
techniques used to yield decision weights and ultimately tne respective 
decision equations. Given the nature of the collection plan, a large 
number of variables (281) resulted for eacli teacher in the sample. 
Logical relations between variables and sets of variables reduced the 
number of potential prea^ctors substantially, flowever, the number of 
logical predictors for the criterion (candidate achievement measure/ 
instructional unit) always exceeded the limit prescribed by the Sc?mple 
size given regression techniques. Consequently, correlation coefficients 
were determined among the variables logically related to the achieve- 
ment score for the instructional component being analyzed. The pro- 
cedure yielded seven potential predictors for each of the components 
cons i dered. 

The second statistical treatment involved the RSQUARE procedure 
from the Statistical Analysis System (SAS) (Barr and Goodnight, 1972). 
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This procedure perfonned mjltiple regressions to the d-pendent variable 

with the seven predictors identified in the preceding step. Optimal 

variance (R =.685) was accounted for by five oF these variables for the 

instructional component dealing with piifornrtnce objectives: 

CT18 - Cooperating teacher ra L"j;^g: relation between tect-objecti ves * j 

US18 - University supervi?.or rating: relation between test- 
objecti ves 

.ICM - CP I scale (communal ity) X.l 

CT35 - Cooperating teacher rating: content o.^ganization 

110 - Follow-up importance rating: lesson planning 

Four variables accounted for the variance (R =^.44) of the component on 

sel f-analysis skills: 

.ISA - CIP scale ( sel f assurance) X.l 

CT30 - Classroom teacher rating: classifying questions 

18 - Follow-up importance rating: leading discussions 

119 - Follow-up importance rating: using different instructional 
modes 

These respective predictor-dependent variable sets for the performance 
objectives and sel f-analysis ski lis instructional components were sub- 
sequently placed in prediction models and analyzed by a stepwise regression 
analysis procedure called REGR in SAS (1972) Results of these analyses 
are presented in Tables 1 and 2. 



Insert Tables 1 & 2 



The F values for both regression models were determined to be significant, 
i.e., F=13.02, p. 0001 (performance objectives), and F=4.73, p. 006 (self- 

ERIC i 0 
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analysis skills). The regression procedure for each prediction model 
produced partial su. ; of squares for each decision variable which were 
employed as dr:: r.wn weights for the variables in the resulting decision 
equations wh^;e :he algebraic sign of each decision weight was obtained 
from the sign of the beta value for that variable. Substituting these 
values into a linear expression, the decision equation for the performance 
objective component became: 

Min Y . 1.11 . 1.01 Xj^Q . 1.40 X^^.g - .S5\^^^ - .32X 

and the decision equation for the self-analysis skill component assumed 
the form: 

Min Y.l. 26X^,30.. 86X ^3^. 1.55Xj^9-1.34Xj3. 

Substituting the mean values (Table 3) for each variable in each 

equation and performing the aritfiiiietic operations yielded 11.48 for the 

If 

performance objective instructional component and 8.27 for the self- 
analysis skill instructional component. These values were compared 
with criterion values (Min Y) calculated for each equation based on 
maximum values for each decision variable multiplied by a .85 accomplish- 
ment factor for the group of teachers. The resulting criterion values 
for the performance objectives component was 11.44 v/hile the cut-off 
value for the self-analysis skills component was 8.77. 

The calculated value for the self-analysis component, (8.27) did 
not reach the criterion value of 8.77. Conversely, the calculated value 
(11.48) for the performance objective component exceeded the cut-off 
value of 11 .44. 



Insert Table 3 
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Discussion 

A cursory examination of the results of these analyses suggests 
that the goals of this project were achieved. However, a number of 
assumptions, observations, and decisions were made which heretofore have 
not been addressed. First, curriculum evaluation as practiced here 
assumed the performance objective to be the basic organizational elen)ent 
in curriculum design. Instructional activities and assessments were 
directly reloted to performance objectives in the various instructional 
components, allov'-zing these curricular elements to be isolated for 
eval uation. 

Second, another assumption of this project was that candidate 
achievement data obtained from criterion-referenced tests could serve 
dual functions. One function of the data was to provide course progress 
indicators for the candidate's course grade, while the second function 
was to evaluate the effectiveness of instructional components related to 
particular objectives. In the case of student progress, assessment 
data were treated idiographi cal ly , that is, the candidate was the unit 
of analysis. However for the second function, program evaluation, the 
achievement data were treated norniatively . This data set was thought to be 
most appropriate for the criterion variable used in deriving the weights 
for the decision equations because of the relation of these data to the 
instructional components and because of the careful attention and tliouglit 
afforded the tests by the candidates. Moreover, Scriven's position on 
payoff evaluation (1967, pp. 59-62) lends credence to the application 
of an achievement data set as the criterion variable in the decision process. 

Third, considerations for weighting the data ranged from intuitively 
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assigning decision weights to devising a multi-step procedure which pro- 
vide empirical justi fication for the ass igned wei ghts . Since tlie data 
collection procedures employed yielded numerous variables and the cri- 
terion variable was known, regression procedures were considered viable 
for quantifying the "weighting" process. Regression procedures are 
relatively free of Operational assumptions, and may be readily employ- 
ed given appropriate computer software, namely, user oriented statis- 
tical packages. Further, stepwise regression analysis yields two 
necessary elements in weighting tlie data: 

a) the partial sums of sqjjare s value indicates tlie unique contri- 
bution of a variable to tlie overall variance in the regression 
model (Draper, Smith, 1966). 

b) beta values for each variable, being vector quantities, repre- . 

sent both spatial orientation and magnitude for that variable 

with respect to the axis of the linear model produced. 

Fourth, the generalized first order linear equation, 
i _ 

Y min 1 I^- X., for evaluating the curriculum components resulted 
after considering whether transformations of the data sets (reciprocal, 
logrithmic, square root, and higher order models [X*^]) would enhance the 
decision equations. For these adjustn^ents, the appropriate clioice should 
be made on the basis of previous knowledge about the effect of the tran:^- 
formation on the variable. Usually, tliis condition is not knowri, for 
example: What is the instructional significance of using tlie natural 
logarithm (In^) of a supervisor's numerical rating? Or how doeS" this 
transformation influence the statistical analysis? These questions illus- 
trate potential unknown effects of possible data transformations. In 
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addition, the linear regression which yields the decision weights will not 
be statistically significant if the first order model digresses too far 
from linearity. Because of these observations, the decision was made to 
use the generalized expression with first order variables rather than 
resort to data transformations which would increase the coniDlexity of 
^the equations. 

Fifth, although high-inference instruments were used in the collection 
plan, especially in phases three and four, it was assumed that data 
obtained from observation scales completed by student teaching supervisors 
and self- reports from former teaching candidates would provide stable and 
valid data for curriculum evaluation. In support of this assumption, 
Turner (1974) has suggested that the perspectives of recent graduates, 
tempered by the real world, are important data sources for evaluating 
the effectiveness of teacher preparation programs. 

Finally, the relations of decision variables to the criterion were 
limited to the range of observations from which the decision weights 
were derived. The significance of this observation is that each time a 
curricular component is to be evaluated a unique decision equation must 
be developed. 

In view of these assumptions, observations, and limitations of the 
evaluation process, one may question whether the project goals were 
actually achieved. To resolve this concern, reconsider the primary goal 
of the project and the steps taken to accomplish it. Essentially, the 
goal called for the mathematical integration of weighted data into deci- 
sion equations for curricular evaluation. The following procedures were 
developed to accomplish this goal: 
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1) Identi fying potentia l decision variable s - This step is accom- 
plished by determining correlation coefficients between the 
array of predictor variables and the criterion variable. Vari- 
able selection is based on the magnitude of the coefficient and 
the logical relation between the variables. 

2) Selecting decision variables - A forward regression procedure 
incorporating the variables identified in step one is performed 
with the criterion variable to determine the "optimal conbina- 
tion" of decision variables. This "optimal conbination" depends 
on the axiom, maximum variance with minimum variables. 

3) Determining deci sion - wei ghts - A stepwise r..^gression pro- 
cedure is conducted with the decision variables identified in 

Step 2. This procedure yields an overall F-test for tlie regression 
model, as well as F values for each decision variable in the 
model. If the overall F value is not significant, indicating 
the variance accounted for by the decision variables is slight 
or the regression is not linear, the procedure to determine a 
decision equation terminates for the instructional component 
under consideration. Conversely, if the overall F value is 
significant, the decision v;eight for each variable assumes tlie 
numerical value of tlie partial s ufns of squares for that 
variable, and the directionality (arithmetic sign) of the 
corresponding beta value. 

4) Incorporating decis ion wei ghts into a^ d ecision e quation " - The 
decision weights resulting from the stepwise regression analysis 
are then substituted into the general expression 
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Y min 1 ^^^i X. to complete the decision equation. 

5. Solving decision equation - The expression on the right side of 
the equatiori is solved by adding the products' of the respective 
decision weights - variable means together" The expression 

Y min is determined in much the sanie manner except maximum values 
replace the mean values. The resulting sum is then multipled 

by a .85 accomplishment factor. This value was selected as the 
accomplishment factor because of potential positive bias on 
rating scales and perception instruments, and the intuition 
that an instructional program should not be considered effec- 
tive unless it is compared with a rigorous but attainable 
standard. Interpretation of the results of these calculations 
depend on the relative magnitudes of the solutions; if the 
value of Y min exceeds the value on the right side- of the ex- 
pression revision of the instructional component should be 
seriously considered. 
That these steps represent a functional process depends on whether 
implementation has occurred and yielded results which are nieaningful. 
Results of the evaluations included in this paper have illustrated that 
indeed these steps are feasible and do provide empirical support for 
curriculum decisions. In essence, this project has operati onal i zed the 
integration of longitudinal data sets into a generalized mathematical 
expression for rendering precise curricular decisions. 
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Table 1 

Stepwise Regression Procedure for the Dependent Variable: 
Learner Achievement on the Performance Objectives Instructional Component 



Source DF SS MS F Prob R' 



RGgression 5 5.989 1.198 13.019 .0001 .685 

Residual 30 2.750 0.092 

Total 35 8.750 



Source Partial SS B value 



CTIB 0.650 -0.276 

US18 1.402 0.429 

-ICM 0.323 -0.051 

CT35 1.106 0.344 

110 1.014 0.162 



CT18 Cooperating Teacher rating: test-objectives relation 

US18 University Supervisor rating: test-objectives relation 

.ICM CP I Scale: Communal ity X.l 

CT35 Cooperating Teacher rating: content organization 

no Follow-up importance rating: lesson planning 
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Table 2 

Stepwise Regression Procedure for the Dependent Variable: 
Learner Achievement on the Self-Analysis Skills Instructional Component 



Source Of SS MS 



Regression 4 8.175 2.044 

Residual 24 10.377 0.432 

Total 28* 18.552 



Source Partial SS B value 



•ISA 0.861 

CT30 1.260 

18 1.335 

119 1.547 



* Missing data reduced the sample of this regression. 

.ISA CPI Scale: Self Assurance X .1 

CT30 Classroom Teacher Rating: Classifying questions 

18 Follow-up importance rating: leading discussions 

119 Follow-up importance rating: using different instructional modes 



F Prob 
4.727 .006 .441 



0.055 
0.338 
-0.206 
0.305 



ERIC 



18 



Mathematical Equations. . . 17 



Table 3 

Sample Means for Independent Variables Used in Decision Equations 



Instructional Components 
Performance Objecti ves Self-Analysis Skills 



Variable X Variable X 

ens 4.53 .ISA 2,r6'^ 

US18 4.48 CT30 4.34 

.ICM 2.63 IB 4.28 

CT35 4.52 119 . 4.28 

no 3.93 
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